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One  of  the  things  that  dismays  people  unfamiliar  with  Modelling  and  Simulation  or  M&S  is  that  we  have  no 
perfect  solution.  Instead,  we  ask  copious  questions  to  get  as  close  as  possible  to  the  objectives  sought. 
Analysis  supporting  ill-defined  requirements  can  be  equally  disastrous  for  the  same  reasons.  Like  M&S, 
the  raison  d'etre  of  analysis  is  built  upon  knowing  what  you  want  to  capture  and  pre-determining  a 
methodology  to  gauge  whether  you  are  getting  it  or  not.  Developing  a  simulation  and  exercise  collection  plan 
therefore  means  two  things  where  the  training  audience  is  concerned:  the  simulation  plan  is  unseen  but  the 
presence  of  a  collection  plan  is  mildly  evident. 

The  dilemmas  with  collection  plans  begin  at  their  initial  inception.  If  physical  observation  is  necessary, 
then  there  must  be  a  focus  since  a  prolonged  presence  interferes  with  a  staff  interacting  within  their  habitat. 
This  requires  research.  Unfortunately  when  an  officer  receives  a  task  for  this  mission,  a  senior  personage  or 
mentor  has  not  even  been  appointed  much  less  identified.  This  impacts  critical  planning  and  time  management 
assuming  that  no  professional  organisation  is  dedicated  for  this  purpose.  Using  an  American  figure  of  speech, 
having  something  ‘fall  in  your  lap’  means  that  this  is  given  to  you  whether  you  were  expecting  it  or  not. 
Assuming  this  is  you,  my  recommendation  is  to  look  at  the  following  critical  areas:  identify  those  receiving 
the  initial  taskings;  look  at  the  training  objectives;  look  at  the  doctrine  in  relation  to  these  objectives. 

What  proves  insightful  is  to  compare  what  is  being  attempted  with  the  overall  training  strategy  of  the 
exercising  unit.  Is  this  headquarters  being  asked  to  do  something  that  it  never  prepared  for?  Further,  by  taking 
a  look  at  the  order  from  higher  and  comparing  it  with  the  headquarters  order,  there  will  be  instances  where  the 
continuation  of  actions  are  not  complete  nor  nuances  from  the  senior  command  fully  embraced.  Before  the 
appointment  of  a  senior  mentor,  it  is  perfectly  acceptable  to  record  any  potential  discrepancies  for  future 
consumption.  There  is  nothing  wrong  with  looking  at  the  time-sequence  of  certain  events  during  the  exercise 
and  creating  a  draft  prioritising  the  events  to  be  observed.  It  is  also  acceptable  to  stake  an  early  claim  over 
observer  space  and  configuration  as  well  as  stating  requirements  for  the  set-up  of  an  After  Action  Review  or 
facility.  Ideally  this  results  from  a  physical  reconnaissance  of  the  exercise  facilities  in  advance. 

Aside  from  the  physical  room  requirements  of  the  team,  there  are  budgetary  issues  that  must  be  addressed. 
Team  members  will  need  to  travel  to  the  event  and  return  home  but  what  happens  while  they  are  there? 
If  observer  controllers  are  treated  as  an  afterthought,  they  will  have  to  travel  farther  if  all  the  good  rooms  in 
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town  are  taken.  In  many  cases,  an  observer  needs  to  travel  from  the  headquarters  to  the  exercise  facility  to 
look  at  both  sides  of  a  communication  or  capture  a  critical  event  from  a  screen  to  accentuate  a  key  point 
during  an  AAR.  There  is  also  a  need  to  discuss  the  navigation  of  trails  and  roads  with  the  installation  so  that 
highly  travelled  areas  such  as  tank  trails  are  avoided,  affording  another  quicker  alternative  route  between  the 
two  points.  While  these  observers  are  collecting,  there  must  be  communications  whether  it  is  mobile  voice  or 
via  network  to  send  briefings  and  files. 

There  are  many  forces  at  work  to  compose  the  observer  team  while  you  are  taking  your  initial  steps.  A  Senior 
Mentor  is  usually  hand  picked  by  the  Commanding  General  or  his  Deputy.  Assuming  that  you  are  the  Senior 
Observer  Controller,  there  is  a  phasing  out  of  future  duties  so  it  does  not  conflict  with  the  training  event. 
Further,  there  are  functional  experts  that  must  be  equally  tasked  so  that  many  of  the  headquarters  core 
competencies  can  be  observed  and  noted  for  a  more  holistic  analysis. 

So  how  will  you  get  there?  It  varies.  In  some  cases,  behaviour  must  be  observed  or  qualitative  answering 
subjectively  how  well  something  was  done.  For  others,  there  are  quantitative  metrics  that  implies 
effectiveness  answering  how  far,  how  fast  or  how  many.  Some  events  have  an  elaborate  collection  scheme 
where  nothing  is  concealed  such  as  the  National  Training  Center  in  California’s  Mojave  Desert  where 
observation  is  conducted  both  by  satellite  and  physical  observation.  There  is  also  a  functional  scheme  where 
rotary  flight  in  an  artificial  environment  is  the  only  thing  observed.  Some  of  the  training  objectives  will 
require  a  combination  of  the  above  -  underscoring  the  need  to  get  started  in  preparatory  analysis  from  the 
moment  you  are  tasked. 

By  the  time  a  senior  mentor  is  appointed,  having  the  physical  layout  of  the  facility  with  observer  controller 
spaces,  an  overlay  with  the  communications  structure,  a  unit  training  schedule  and  exercise  order  is  a  good 
start.  Using  a  ‘draft’  collection  plan,  a  senior  mentor  will  be  hard  pressed  to  alter  much  of  what  you’ve 
prepared  since  you  have  addressed  meetings,  observation  templates  and  critical  observation  tasks  based  on  the 
materials  you’ve  obtained.  If  a  senior  mentor  makes  any  adjustment  at  all,  it  is  the  technique  in  which  the 
observations  are  made.  Nonetheless,  once  the  senior  mentor  has  made  those  adjustments,  a  collection  plan  is 
ready  to  publish  within  your  team. 

Other  adjustments  will  have  to  be  made  closer  to  the  event  because  the  plan  changes.  A  General  Officer  may 
dictate  a  unit  going  right  instead  of  left  or  incorporate  ‘innovations’  such  as  the  exercising  headquarters 
physically  relocating  simultaneous  to  a  mission.  This  means  that  more  than  one  observation  technique  must  be 
used,  requiring  someone  to  maintain  equipment  such  as  video  and  digital  cameras  ready  at  all  times. 

After  Action  Reviews  or  AARs  are  reinforced  when  the  training  audience  see  themselves  in  relation  to  what 
they  were  trying  to  accomplish.  This  places  the  focal  point  of  the  AAR  off  of  your  team  and  on  them.  If  AARs 
are  interim  it  occurs  between  missions  and  therefore  must  be  respectful  of  the  tight  schedule  placed  on  the 
training  audience.  An  informal  AAR  giving  cursory  feedback  on  how  they  did  is  normally  sufficient.  In  all 
cases,  the  end  of  exercise  requires  a  formal  AAR.  This  is  because  the  concentration  of  all  exercise  participants 
is  directed  on  what  you  have  done  from  beginning  to  end.  Despite  the  fact  that  many  people  are  thinking  of 
returning  home,  there  are  two  categories  of  attendees  that  will  really  care  what  you  have  to  say:  those  that  are 
Generals  and  those  that  hope  to  be. 

To  leave  a  lasting  impression,  a  Take  Home  Package  or  THP  takes  all  written  matter,  photography, 
screenshots  and  statistical  tables  and  indexes  the  points  learned  during  the  training  event.  Coupled  with  the 
AAR  videotape,  this  THP  represents  a  crisp,  professional  effort  befitting  years  of  future  study.  By  keeping  a 
copy  of  your  own  product,  you  will  see  the  stark  difference  between  your  efforts  and  all  else  that  follows. 
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Learning  Objectives 


Provide  training  audience  with  performance  feedback. 

Develop  a  simulation/exercise  collection  plan 
Explain  the  process  of  collecting  exercise  events 
for  feedback  purposes 
Describe  the  AAR  process 
Explain  the  final  exercise  report  process 


Putting  Your  Best  Foot  Forward 
When  It  Falls  in  Your  Lap 


EVALUATE: 

•  TASKINGS 

•  DOCTRINE 

•  TRAINING  OBJECTIVES 

•  ORDERS 

•  MISSION  ESSENTIAL  TASKS  LIST 

•  TIME-SPATIAL  SEQUENCING  (RIGHT  PLACE,  RIGHT  TIME) 

•  CONDUCTING  AFTER  ACTION  REVIEWS 


Making  the  Most  of  an 
Eight-Hour  Day 


PRIORITIES  OF  WORK: 

•  SENIOR  MENTOR  (CHIEF  OF  STAFF/CG) 

•  SENIOR  OC(J3/CHIEF  OF  STAFF/CG) 

•  OBSERVER  CONTROLLER  TEAM  (J3) 

•  FUND  CITE  (COMPTROLLER) 

•  SIM  CENTER  (INSTALLATION) 

•  VEHICLES/COMMO  FOR  OFF-ROAD  (J3) 

•  LODGING  (INSTALLATION) 

•  AUTOMATION  (J6/INSTALLATION) 

•  CELL  PHONES  (J6/INSTALLATION) 

•  RANGE  CONTROL  (INSTALLATION) 


The  AAR  Collection  Plan 


Questions  to  Ask: 

Who  will  observe  the  training  and  who  will  conduct  the  AAR? 

What  trainers  should  evaluate  (training  and  evaluation  outlines  -  TEOs)? 
Who  is  to  attend? 

When  and  where  the  AAR  will  occur? 

What  training  aids  trainers  will  use? 

Quantitative  analysis?  (Engagement/Kills,  Red-Amber-Green,  etc) 
Qualitative  analysis?  (“This  is  what  I  observed”) 


Elaborate  Data  Collection 


NTC  INSTRUMENTATION 


INTEGRAL  PART  OF  DATA  COLLECTION  EFFORTS 
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Quantitative  vs.  Qualitative 
Techniques 


What  are  the  major  differences  between  quantitative  and 

qualitative  techniques? 


Value  of  the  data 


Quantitative  and  qualitative  techniques  provide 
a  tradeoff  between  breadth  and  depth 


Generalizing  and  targeting  to  specific 
(sometimes  very  limited)  populations. 


Quantitative  vs.  Qualitative 
Techniques 


What  are  the  major  differences  between  quantitative  and 

qualitative  techniques? 


Philosophical  Distinction 


Qualitative  researchers  argue  that 
there  is  no  objective  social  reality,  and 
that  all  knowledge  is  "constructed"  by 
observers  who  are  the  product  of 
traditions,  beliefs,  and  the  social  and 
political  environment  within  which  they 

operate. 


Quantitative  researchers  no 
longer  believe  that  their 
research  methods  yield 
absolute  and  objective  truth. 

They  continue  to  adhere  to  the 
scientific  model  and  seek  to 
develop  increasingly 
sophisticated  techniques  and 
statistical  tools  to  improve  the 
measurement  of  social 
phenomena. 


Quantitative  vs.  Qualitative 
Techniques 


What  are  the  major  differences  between  quantitative  and 

qualitative  techniques? 


Philosophical  Distinction 


Qualitative  approach  emphasizes  the  importance  of  understanding 
the  context  in  which  events  and  outcomes  occur. 

Quantitative  researchers  seek  to  control  the  context  by  using  random 
assignment  and  multivariate  analyses. 


Qualitative  researchers  believe  that  the  study  of  deviant  cases 
provides  important  insights  for  the  interpretation  of  findings; 

Quantitative  researchers  tend  to  ignore  the  small  number  of 
deviant  and  extreme  cases. 


Quantitative  vs.  Qualitative 
Techniques 


What  are  the  major  differences  between  quantitative  and 

qualitative  techniques? 


Scientific  Rigor 


Quantitative  methods  believed  to  yield  more  objective,  accurate  information 
because  data  collected  using  standardized  methods  and  can  be  replicated. 

Unlike  qualitative  data,  can  be  analyzed  using  sophisticated  statistical 
techniques. 


Qualitative  methods  are  most  suitable  for  formative 
evaluations,  whereas  summative  evaluations  require  "hard" 
(quantitative)  measures  to  judge  the  ultimate  value  of  the 

project. 


Quantitative  vs.  Qualitative 
Techniques 


What  are  the  major  differences  between  quantitative  and 

qualitative  techniques? 


Time 

constraints 


Staff  skills 


Costs 


Credibility  of 
findings 


Quantitative  vs.  Qualitative 
Techniques 


Why  use  a  mixed  method  approach? 


When  investigating  human  behavior  and  attitudes,  it  is  most 
fruitful  to  use  a  variety  of  data  collection  methods. 


Assumption  -  most  interventions  are  not  introduced  into  a  sterile  laboratory, 
but  rather  into  a  complex  social  environment  with  features  that  affect  success. 


By  using  different  sources  and  methods  at  various  points  in  the  evaluation  process, 
the  evaluation  team  can  build  on  the  strength  of  each  type  of  data  collection  and 

minimize  the  weaknesses  of  any  single  approach. 


Multi-method  approach  to  evaluation  can  increase  both  the  validity  and 

reliability  of  evaluation  data. 


Handing  Off  to  the  Senior  Mentor 


•  A  Map  or  room  schematic  for  the  training 

•  An  overlay  with  the  OC  communications  systems  and  meeting  places  (field) 

•  A  published  training  schedule 

•  The  collection  plan: 

•  The  0600/1800  meetings  (priorities  of  the  day/significant  events) 

•  The  observation  sheets,  “three-up,  three-down”,  Red-Amber-Green 

•  The  critical  tasks  matrix 


Exercise  Sample  Schedule 


Schedule  for  Exercise 

Nov  23  Exercise  Starts  -  Welcome  ceremonies 
Nov  24  Seminars  for  military  staff  procedures 
Nov  25  Information  management  training 
Nov  25  COMEX 

Nov  26  Mini  Ex,  as  preparations  to  final  execution  of  the  exercise 

Nov  27  Exercise 

Nov  28  Exercise 

Nov  29  Exercise  Visitors  day  #  1 

Nov  30  Exercise  Visitors  day  #  2 

Seminars  on  MULTIFUNCTIONAL  PEACEKEEPING 

Dec  01  SIMEX 

Seminars  on  MULTIFUNCTIONAL  PEACEKEEPING 

Dec  02  SIMEX  ends 

Lessons  learned  Farewell  ceremonies 

ENDEX 


Observational  Techniques 


Observational  techniques 

Methods  by  which  an  individual  or  individuals  gather  firsthand  data  on 

processes  or  behaviors  being  studied. 


They  provide  evaluators  with  an  opportunity  to  collect  data  on  a  wide 
range  of  behaviors,  to  capture  a  great  variety  of  interactions,  and  to 

openly  explore  the  evaluation  topic. 


1 

When  to  use  observations 

Observations  can  be  useful  during  both  the  formative  and 
summative  phases  of  evaluation. 

J 


Observational  Techniques 


Observational  techniques  -  advantages 

•  Provide  direct  information  about  behavior  of  individuals  and  groups 

•  Permit  evaluator  to  enter  into  and  understand  situation/context 

•  Provide  good  opportunities  for  identifying  unanticipated  outcomes 

•  Exist  in  natural,  unstructured,  and  flexible  setting 


Observational  techniques  -  disadvantages 

•  Expensive  and  time  consuming 

•  Need  well-qualified,  highly  trained  observers;  may  need  to  be  content  experts 

•  May  affect  behavior  of  participants 

•  Selective  perception  of  observer  may  distort  data 

•  Investigator  has  little  control  over  situation 

•  Behavior  or  set  of  behaviors  observed  may  be  atypical 

•  Recording  observational  data 

•  Comes  to  the  scene  with  a  set  of  target  concepts,  definitions,  and  criteria  for 

describing  events. 

•  Observations  usually  are  guided  by  a  structured  protocol. 

•  A  protocol  helps  assure  that  all  observers  are  gathering  the  pertinent  information  and 

applying  the  same  criteria  in  the  evaluation. 


The  Collection  Plan 
for 

November  26th 


0600:  Shift  Change/  Staff  Update 

Battle  Damage  Assessment  Brief 
0800:  Orders  Brief 

National  Support  Update 
1000:  Estimated  Air  Campaign  Completion 
VTC  with  AFSOUTH 
1200:  Visit  from  PM 

Press  Conference 
1400:  Fires  Coordination  Update 
Fly  to  1 st  Div 
1600:  Fly  to  2d  Div 

Return/  Intel  Update 
1800:  Shift  Change/  Staff  Update 
Issue  FRAGO 

2000:  Estimated  Beginning  Phase  III 
Meeting  with  NGOs 
2200:  VTC  with  AFSOUTH 

Commence  OPERATION  LUCKY 
0001:  Reports  to  AFSOUTH 


0600:  MAJ  Martino  (Night  Shift) 

Observer  Huddle  (COL  DeSilva) 

0800:  LTC  Mattina/MAJ  Pieters  attend 
MAJ  Padrewski  attend 
1000:  Begin  BDA  Analysis 
COL  DeSilva  attends 
1200:  Review  staff  cell  traffic 

Colonel  DeSilva  Departs  to  Heliport 
1400:  Colonel  DeSilva  Flies  to  1st  Div 

Review  Fire  Coordination/Intel  Update 
1600:  Colonel  DeSilva  Returns  from  2d  Div 
with  CG 

1800:  LTC  Mattina/MAJ  Pieters  attends 
Exercise  Control  Meeting 
2000:  Colonel  DeSilva  calls  OC  Cell  AFSOUTH 
Observer  Huddle 

2200:  MAJ  Martino  -  Office  Call  with  Exercise 
Director 

0001:  MAJ  Martino  reads  reports 


Documenting  the  Interactions 


Role  of  the  Observer 


Observer  stands  apart  from  the  setting,  attempts  to  be  nonintrusive, 
and  assumes  the  role  of  a  "fly-on-the-wall." 


May  be  beneficial  to  have  two  people  observing  at  the  same  time. 


A  special  issue  relates  to  the  amount  of  observation  needed. 

There  is  no  hard  and  fast  rule  regarding  how  many  samples  need  to  be  drawn. 

Interviews  provide  different  data  from  observations.  They  allow  the 
evaluation  team  to  capture  the  perspectives  of  project  participants,  staff, 

and  others  associated  with  the  project. 

Structured  interviews,  in  which  a  carefully  worded  questionnaire  is  administered. 

In-depth  interviews,  in  which  the  interviewer  does  not  follow  a  rigid  form. 

Documenting  the  Interactions 


Field  notes 

Date  and  time  of  the  observation  should  be  recorded,  and 
everything  that  the  observer  believes  to  be  worth  noting 
should  be  included.  No  information  should  be  trusted  to 

future  recall. 


Technological  tools 


Tape 

recorder 


Laptop 

computer 


After-action  reviews: 

Are  conducted  during  or  immediately  after  each  event. 
Focus  on  intended  training  objectives. 

Focus  on  soldier,  leader,  and  unit  performance. 

Involve  all  participants  in  the  discussion. 

Use  open-ended  questions. 

Are  related  to  specific  standards. 

Determine  strengths  and  weaknesses. 

Link  performance  to  subsequent  training 


AAR  101 


What’s  involved: 

Introduction  and  rules. 

Review  of  training  objectives. 

Commander's  mission  and  intent  (what  was  supposed  to  happen). 

Opposing  force  (OPFOR)  commander's  mission  and  intent  (when  appropriate). 
Relevant  doctrine  and  tactics,  techniques,  and  procedures  (TTPs). 

Summary  of  recent  events  (what  happened). 

Discussion  of  key  issues  (why  it  happened  and  how  to  improve). 

Discussion  of  optional  issues. 

Discussion  of  force  protection  issues  (discussed  throughout). 

Closing  comments  (summary). 


The  Dichotomy  of 
Formal  Versus  Informal 


Formal  reviews-- 

Have  external  observers  and  controllers  (OCs) 
Take  more  time. 

Use  complex  training  aids. 

Are  scheduled  beforehand. 

Are  conducted  where  best  supported. 

Informal  reviews- 

Conducted  by  internal  chain  of  command. 

Take  less  time. 

Use  simple  training  aids. 

Are  conducted  when  needed. 

Are  held  at  the  training  site. 


Art  and  Science  of  the  AAR 


Conduct  of  the  Evaluation 


Setting 

The  physical  environment  within  which 
the  project  takes  place. 


Human,  social  environment 

The  ways  in  which  all  actors  (staff, 
participants,  others)  interact  and 
behave  toward  each  other. 


Project  implementation  activities 

What  goes  on  in  the  life  of  the  project? 

What  do  various  actors  (staff, 
participants,  others)  actually  do?  How 
are  resources  allocated? 


Conduct  of  the  Evaluation 


Native  language  of  the  program 

Different  organizations  and  agencies  have  their 
own  language  or  jargon  to  describe  the  problems 
they  deal  with  in  their  work.  Capturing  the  precise 
language  of  all  participants  is  a  way  to  record  how 
staff  and  participants  understand  their 
experiences. 


Notable  nonoccurrence 

Determining  what  is  not  occurring  although 
the  expectation  is  that  it  should  be  occurring 
as  planned  by  the  project  team,  or  noting  the 
absence  of  some  particular  activity/factor  that 
is  noteworthy  and  would  serve  as  added 

information. 


Conduct  of  the  Evaluation 


Nonverbal  communication 

Nonverbal  cues  about  what  is  happening  in  the 
project:  on  the  way  all  participants  dress,  express 
opinions,  physically  space  themselves  during 
discussions,  and  arrange  themselves  in  their 

physical  setting. 


Take-Home  Packages  (THP) 


•  Recaptures  the  unit’s  mission,  orders  and  graphics. 

•  Recalls  critical  events  and  observations. 

•  Delineates  an  impartial  analysis  of  the  observer  or  team  of  observers. 

•  The  ‘snapshots’  from  the  model  act  as  a  graphic  representation  of  the 
plan’s  execution. 

•  Incorporates  statistical  analysis  (quantitative)  or  subjective  analysis 
(qualitative) 

•  Includes  the  AAR  videotape. 


Cautions 
Too  lengthy 

Filled  with  unnecessary,  irrelevant,  and  sometimes  inconsistent  details. 
Fail  to  provide  concrete  solutions  for  their  training  problems 


Learning  Objectives 


Provide  training  audience  with  performance  feedback. 

Develop  a  simulation/exercise  collection  plan 
Explain  the  process  of  collecting  exercise  events 
for  feedback  purposes 
Describe  the  AAR  process 
Explain  the  final  exercise  report  process 


